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ABSTRACT 



This document describes the research design that will guide 
the Southwest Educational Development Laboratory's efforts in constructing 
procedural knowledge ("know-how" or working knowledge) needed to transform 
low-performing school districts and schools into high-performing learning 
communities. Little is known about procedural knowledge in this context. The 
research design focuses on three questions: (1) To what extent are 

low-performing districts and schools increasing their capacities to work 
systemically and thereby transform themselves into high-performing learning 
communities? (2) What strategies contribute to increasing the capacities of 
low-performing districts and schools to work systemically and transform 
themselves into high-performing learning communities? and (3) What knowledge, 
skills, and tools are needed by technical-assistance providers to transform 
low-performing districts and schools, and what training is needed to equip 
them to do this work? A minimum of 40 low-performing sites will be selected 
for the study. Methods of data collection and analysis, study limitations, 
and participation incentives are described. Results will be reviewed and 
summarized in a report submitted to the Office of Educational Research and 
Improvement, U.S. Department of Education, annually. An appendix contains 
demographic and contextual information. (Contains 18 references and 1 table.) 
(RT) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 
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Introduction 

This research design will guide SEDL's research efforts in Task 1, Problem 2: to 
construct procedural knowledge to transform low-performing districts and schools into 
high-performing learning communities. Procedural knowledge is the "know-how" 
required to work systemically to transform a low-performing district or school into a 
high-performing learning community. Implicit in the definition of procedural 
knowledge is the capacity to carry out this transformational process (Brophy, 1987; 
Cauley, 1986; McCormick, 1997; Ryle, 1949). However, procedural knowledge also 
requires the expertise to figure out how to apply it in different contexts (Glaser, 1984, 
1992; McCormick, 1997). Kennedy (1983) expands this notion of procedural knowledge, 
suggesting that it is the "working knowledge" that accumulates over time, not static, 
but evolving as new situations or evidence are encountered. 

Relatively little is known regarding the procedural knowledge needed to enable 
low-performing districts and schools to work systemically and transform into high- 
performing learning communities. Given this state of affairs, it is critical to first identify 
which procedures might be most effective in helping low-performing districts and 
schools transform themselves into high-performing learning communities. Once 
identified, through detailed documentation of a number of diverse cases, the 
procedures could then be codified and used as part of a research study that could 
strengthen their identification as effective means to transformation. SEDL proposes in 
this design to carry out the initial study in a sample of between 40 to 60 sites (20 districts 
and 20 to 40 schools) over a five-year period. Once that work is sufficiently advanced to 
identify a coherent set of procedures for working systemically at the district and school 
levels, SEDL will consider subsequent research on the procedures based on an 
intervention approach. 

As part of its Task 1 work, SEDL will partner with these sites to help build their 
capacity to work systemically and, as a result, operate as high-performing learning 
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communities. SEDL will design strategies to be used with appropriate persons at each 
site to support the district's or school's transformation. SEDL will rigorously track the 
implementation of these strategies, documenting them fully, following a 
comprehensive data collection plan. In addition, data will be collected to assess their 
effectiveness or gauge their results. Using the large data base that will accrue, and 
manipulating the data through cross-site analyses, SEDL will determine which 
strategies, under what circumstances, and for what purposes, worked. From such 
analyses, SEDL will be able to suggest how to build the capacity of districts and schools 
to work systemically and become high-performing learning communities (Dentler, 

1984; Fullan & Miles, 1992; Kliebard, 1993; Louis, Dentler with Rosenblum, 1982; 
McCormick, 1997). 

Research Questions 

In order to construct procedural knowledge to support the transformation of 
low-performing districts and schools into high-performing learning communities, the 
design focuses on three interrelated research questions: 

Research Question 1: 

To what extent are low-performing districts and schools increasing their 
capacities to work systemically and thereby transform themselves into high 
performing learning communities? 

Research Question 2: 

What strategies contribute to increasing the capacities of low-performing districts 
and schools to work systemically and transform themselves into high- 
performing learning communities? 

Research Question 3: 

What knowledge, skills, and tools are needed by technical assistance (TA) 
providers to transform low-performing districts and schools, and what training 
is needed to equip them to do this work? 

Research Question 1 

SEDL believes that there is a strong relationship between districts and schools 
learning to work systemically and their transformation into high-performing learning 
communities. Working systemically requires attention to all levels of the system — 
classroom, school, district, and state. It requires attention to all parts or components of 
the system, including standards, curriculum and instruction, assessment, policy and 
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governance, professional staff, resources, and family and community. Further, it 
requires that competencies in working systemically be mastered by the district and 
school staffs (see Refined Action Plan, pp. 6-7). SEDL hypothesizes that as low- 
performing districts and schools develop the capacity to work across the levels, 
components, and competencies (i.e., to work systemically), student achievement (in 
reading and mathematics) will increase, and the districts and schools will take on the 
characteristics of high-performing learning communities. The first research question 
consequently focuses on measuring district and school changes in student achievement 
and status as a high-performing learning community. 

Research Question 2 

Almost all low-performing districts and schools require assistance to increase 
their capacity to work systemically. SEDL will design and implement strategies to assist 
these districts and schools in learning how to work systemically and become high- 
performing learning communities. The systematic documentation and assessment of 
such strategies and their impacts on the transformation of low-performing districts and 
schools will help build SEDL's understanding of how to increase the capacity of districts 
and schools to work systemically and become high-performing learning communities, 
which is the focus of the second research question. 

Research Question 3 

As noted above, low-performing districts and schools require assistance to 
increase their capacity to work systemically. Discussions with the Chief State School 
Officers or their senior staff in SEDL's five-state region revealed that there are not 
enough technical assistance (TA) providers available to provide the necessary high- 
quality assistance to low-performing districts and schools. As a result, there is a clear 
need to learn how to equip TA providers to increase the capacity of districts and schools 
to work systemically and to become high-performing learning communities. SEDL 
proposes to design strategies aimed at equipping TA providers to support such work. 
The third research question, therefore, will focus on the design and assessment of such 
strategies and their impacts on TA providers. 

In brief, then, each of the three research questions addresses different issues 
regarding the construction of procedural knowledge. Research Question 1, in essence, 
addresses the effects that accrue as a result of the work conducted in Research Questions 
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2 and 3. SEDL believes that it is important to "begin with the end in mind" (Covey, 
1998) — and this end is embedded in Research Question 1: working systemically to raise 
student achievement and become a high-performing learning community. How SEDL 
achieves "the end" is reached through the strategies involved in Research Questions 2 
and 3. Therefore, answers to Research Questions 2 and 3 will inform SEDL about how to 
attain the goal of Research Question 1. Table 1 summarizes the research design. The 
subsequent text describes the sample, the data collection and analysis procedures 
proposed to answer each of the research questions, the annual reporting of research 
findings, and the limitations of this study. 
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Sample 

The selection of a minimum of 40 sites (i.e., 20 districts and one or more schools 
per district) that are low-performing in reading or mathematics student achievement 
will be guided by the criteria noted below. These criteria were reviewed by each state's 
Chief State School Officer and/ or senior staff, who subsequently provided advice and 
counsel in the final selection of sites. The sample of sites will reflect the following 
criteria: 

• Different student achievement profiles will be represented, including schools that 
are low achieving in reading and mathematics (as defined by the state's 
particular measure), low in reading only or low in mathematics only; districts 
that have only low-performing schools versus districts that have only a small 
percentage of low-performing schools. 

• Schools at all levels (elementary, middle, high schools) will be included, and 
when multiple schools are included from a district, the schools will be part of a 
feeder system whenever possible. 

• At least 25 percent of the students enrolled in a selected school will be low- 
income (eligible for free or reduced lunch) or classified as limited English 
proficient. 

• At least 25 percent of the sites will be located in rural areas. 

• The superintendent, principals, and faculty will agree to participate in this effort, 
will be committed to all students achieving high standards, and will be prepared 
to alter their educational system. 

• District and school leaders will be willing to actively involve the faculty, staff, 
parents, and community in building a high-performing learning community. 

• District and school resources will be available to support the work. 

• A technical assistance provider from the SEA or intermediate service unit will be 
committed to working on this effort with SEDL which, in turn, will provide 
resources to support these individuals. 



Research Question 1: To what extent are low-performing districts and 
schools increasing their capacities to work systemically and thereby 
transform themselves into high-performing learning communities? 

Rationale 

Districts and schools tend to have unique cultures that develop over time as 
people work together to solve problems and confront challenges. This is why reforms 
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based on assumptions of uniformity in the educational system repeatedly fail. On the 
other hand, successful reform efforts adapt to and capitalize on this contextual 
variability. They shape and integrate reform efforts in ways that best suit their 
individual and organizational context. This suggests a systemic approach to reform 
efforts that is sensitive to contextual differences, and yet one that pushes districts and 
schools to tackle complex problems across the levels and components that exist in the 
education system. It is hypothesized that if districts and schools learn to operate in this 
manner, student performance in reading and mathematics will improve, and districts 
and schools will become high-performing learning communities. 

As noted above, this research question assesses the progress that study districts 
and schools are making in achieving these two outcomes. It is believed that this 
assessment can best be accomplished by dividing Research Question 1 into two sub- 
questions. Sub-Question 1.1 addresses the student performance outcome in reading 
and mathematics, and Sub-Question 1.2 addresses the outcome related to working 
systemically and becoming a high-performing learning community. 

Research Sub-Question 1.1: How has the achievement of students in 
reading and mathematics changed? 

Data Collection 

Changes in student achievement will be assessed using the data for reading and 
mathematics that are derived from state-mandated tests. This will require that we 
collect: 

• test data for reading and mathematics at the school level for each school year; 

• test data for reading and mathematics at the district level for each school year; 

and 

• test data for reading and mathematics at the state level for each school year. 

The first set of data to be used at each of these levels will come from the tests 
administered in the Spring of 2001. SEDL will collect individual student test data for all 
grade levels tested. In some cases, individual student data may not be released. In these 
cases, SEDL will gather test data for individual grade levels. Disaggregated test data for 
different student groupings (e.g., gender, race/ethnicity, socioeconomic status) also will 
be gathered whenever available. 
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Data Analysis 

Several steps will be taken in the analysis of data to determine student progress 
in reading and mathematics. 

Step 1. At the school level, analyses will first be conducted to determine if bias 
exists in the student data set (if individual student test data are available). In each 
school, there will be two subgroups of students: 1) those who have complete test 
scores, and 2) those who do not have complete test scores. It is assumed that the group 
of students with missing test data may represent a potential bias in the sample that 
warrants further investigation. The two subgroups of students will be compared (using 
chi-square tests) to determine if there are statistically significant differences between the 
two subgroups on selected demographic characteristics. If no statistically significant 
differences are found, then subsequent achievement analyses can be conducted with 
confidence that the student achievement data set is not biased. If significant differences 
are found, then appropriate cautions must be taken in analyzing and interpreting 
student achievement data. 

Step 2. At each of the three levels (school, district, and state), statistical analyses 
will be conducted to determine the progress made in reading and mathematics from 
one test period to the next one. This will provide an annual analysis of student 
progress. 

Step 3. In some states, students are tested each year. In these states, each year's 
student test data will be compared with matched data from the initial testing in the 
Spring of 2001 and any subsequent year(s). These comparisons will provide a 
longitudinal assessment of student progress. 

Step 4. The progress of study schools will be compared with the progress of 
other schools at the district and state levels annually and longitudinally (when possible). 
The progress of study schools will also be compared to the progress of "like" schools 
(in the district and state), and the progress of study districts with like districts (in the 
state). By "like schools or districts," it is meant that they would be compared only with 
other schools or districts with similar demographic characteristics and initial 
achievement records. 

In all of the steps described above, the appropriate statistical test will be applied 
to determine the significance of any student gains within the school, the district, and the 
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state. These may include chi-square tests, analyses of covariance, and regression 
analyses, depending on the availability of the necessary data. All analyses will conform 
to any and all sub-analyses of test scores that are typically used at the school, district, 
and state levels. For example, if test scores are analyzed by grade level, then grade 
level analyses will be conducted. Other indicators that are routinely used by the district 
or school to determine school and/or student progress in reading and mathematics will 
be tracked and considered as appropriate. 

Research Sub-Question 1.2: What progress is each district and school 
making toward working systemically and becoming a high-performing 

learning community? 

Data Collection 

A clear description of what it means to work systemically is needed to determine 
the progress that each site is making. This description must then be translated into an 
instrument that can assess movement toward this goal. In the Refined Action Plan (pp. 
6-15), a framework for transforming an educational system into a high-performing 
learning community is presented. This framework includes the levels, components, and 
competencies in the educational system that must be addressed to work systemically. 
SEDL hypothesizes that when a site (a district or school) learns to work systemically, it 
will become a high-performing learning community. The six principles that describe 
the operation of a high-performing learning community are indicated. 

Innovation Configuration Map. SEDL will develop an Innovation 
Configuration Map (ICM) (Hall & Hord, 2001) that encompasses the framework and 
the high-performing learning community principles. (This instrument will be created in 
the second quarter of FY 01.) The ICM will be designed in a format that is easy for 
individuals to complete and for SEDL researchers to analyze. The SEDL coordinator 
assigned to each site will be responsible for administering the instrument. At the school 
level, it will be completed by all personnel in the study school(s). It will also be 
completed by district personnel and SEDL intensive site staff 1 and external TA 

1 Staff from the Charles A. Dana Center, The University of Texas at Austin and American Indian 
Research and Development are working with SEDL in these intensive sites. In the remaining sections of 
the design, SEDL intensive site staff refers to staff from SEDL as well as the Dana Center and AIRD. 
These individuals should not be confused with the SEDL research team who do not have responsibility 
for providing direct support to these sites. 
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providers who are involved in work at the study school(s). At the district level/ the 
ICM will be completed by selected personnel at the district level, and SEDL intensive 
site staff and external TA providers who are working at the district level. The ICM will 
be completed at the beginning of the study and at the end of each school year 
thereafter. 

Interviews. Another data source for assessing progress toward the goal of 
working systemically will be interviews that are to be conducted three times in the first 
year (in the early fall, winter, and late spring). Thereafter, there will be two interviews 
conducted each year, in early winter and late spring. The interviews will be completed 
by the SEDL research team. Included in the interviews will be the district 
superintendent, school principals, external TA providers who work along side SEDL 
intensive site staff, and a sample of teachers, board members, parents, students, and 
community members. The interview questions will parallel as much as is feasible the 
information contained in the ICM, thus triangulating and enriching the ICM data. (A 
specific interview protocol will be developed in the second quarter of FY 01.) After 
completing the late spring round of interviews, the SEDL research team who conducted 
the interviews for a particular district or school also will complete an ICM. This will 
provide an additional source for assessing the progress of a site in learning to work 
systemically and becoming a high-performing learning community. 



Data Analysis 

The ICM permits a quantitative comparison of resulting data. For example, for 
each component or each dimension of the ICM, respondents will be given a range of 
options for describing where they perceive the site (district or school) to be at that time. 
Each option will be assigned a value, individual responses will be scored, and a mean 
value determined for the district or for the school. Over time, changes in the mean 
values assigned to each dimension will be compared as a way of determining progress 
toward the goal of working systemically. The data from the ICM will also be subjected 
to a cluster analysis (George, Hall, & Uchiyama, 2000) that will group districts and 
schools into high, medium, or low groups based on their ICM values. Both the mean 
values and the cluster data will be related quantitatively to student achievement (at the 
district or school levels) to examine the relationship between student achievement and 
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the progress of the districts or schools in learning to work systemically and becoming 
high-performing learning communities. 

The ICMs completed by the district and school personnel, SEDL intensive site 
staff, and TA providers will be compared to the ICMs completed by the SEDL 
researchers to triangulate the findings from the former. Data from the interviews also 
will be analyzed to triangulate the findings from the ICM. Selected software (under 
exploration at this time) will be used to permit entry of interview data into the 
computer database that will make statistical comparisons of the quantifiable interview 
data with the ICM data possible. In addition to the statistical analyses described above, 
data from the ICM and the interviews will be carefully scrutinized by SEDL for trends 
and anomalies within the data set. 

Year 1 Timeline 

September 2001 

Early Fall 
Early Winter 
Late Spring 
May 2002 

June to Aug 2002 

The spring administration of the ICM and the conduct of the interviews will be repeated 
in the second through fifth years of the contract according to the same schedule, except 
that only two rounds of interviews will be conducted in the second through fifth years 
of the contract, in winter and spring. 



Administer ICM instrument (or as early as possible) 

Access test data from reading and mathematics for Spring 2001 

Conduct first round of interviews 

Conduct second round of interviews 

Conduct third round of interviews 

Administer ICM instrument 

Access data from end-of-year reading and mathematics tests (this 
will be done when it is made available) 

Organize, analyze, and interpret data 
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Research Question 2: What strategies contribute to increasing the 

capacities of low-performing districts and schools to work systemically 
and transform themselves into high-performing learning communities? 

Rationale 

This research question addresses the documentation of strategies taken to build 
the capacity of districts and schools to work systemically and transform themselves into 
high-performing learning communities, and the assessment of these strategies' 
effectiveness. Of great interest will be the levels, components, and competencies of the 
framework that will anchor the transformation work and the strategies developed and 
implemented to assist districts or schools in carrying out their work. An understanding 
of these strategies and their intents, and the contexts in which they are successful in 
districts and schools, is essential to comprehending the transformation process. 

Data Collection/Instruments 

Three primary data collection strategies will be utilized to address this research 
question: running logs kept by site coordinators; the collection of data related to the site 
work by site coordinators and other intensive site staff; and interviews conducted by 
the research team with key representatives of the districts and schools. These 
interviews will also address issues raised in Research Sub-Question 1.2 and Research 
Question 3. The instruments (logs, site records, and interview protocols) will be created 
in the second quarter of FY 01. 

Logs. Each site will have a SEDL staff member assigned as site coordinator. This 
individual will maintain a running log of all contacts (e.g., phone, e-mail, site visits, 
letters) at the site. These logs will document the date of the contact, the type of the 
contact, the source and target of the contact, and the specific location where it took 
place. The logs will help to verify that appointments, meetings, and other activities 
occurred at the sites. The entries are meant to provide a brief recording of the 
transformation-related work at the particular site. The logs will be maintained 
electronically in searchable files with a password system that SEDL intensive site and 
research staff and TA providers can access. 

Site Contact Record. Field notes will be kept by SEDL intensive site staff and/ or 
external TA providers who are involved in the site work. These notes will be recorded 
in a Site Contact Record, which includes two sections. In the first section, Section A, the 
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recording will focus on the transformation work that is undertaken with the site 
(district or schools). Whenever work is done at a district or school, the SEDL staff 
member or TA provider will enter the data into an electronic site record that will be 
available (through password access) to SEDL intensive site and research staff members 
and TA providers for review and study. 

SEDL will code specific aspects or dimensions of the transformation work at each 
site. These will include: 

• date of the specific action; 

• site name; 

• name and role of individual recording the notes; 

• a statement of the strategy and its intent and purpose; 

• a brief statement of the action; 

• who did the action; 

• the target of the action (person or persons); 

• where the action took place; 

• the length of time required by the action; and 

• which of the levels, competencies, components, and principles were addressed. 

In Section B of the Site Contact Record, the field notes will be expanded to obtain 
a rich, in-depth narrative description of what occurred. This narrative description will 
provide the foundation for constructing the procedural knowledge for practitioners. It 
will describe the particular strategy and actions taken at the site and their outcomes. It 
will discuss the intent (or purpose of the strategy), why this strategy was selected 
(rationale), and how the actions relate to the particular strategy. The narrative will 
document the framework levels, components, and competencies that were addressed, 
and their relationship to the underlying high-performing learning community 
principles. It will also include reflections and insights related to the particular strategy 
and actions that took place. Issues for additional consideration, next steps, and other 
relevant information will also be recorded. 

Interviews. The interviews will be conducted with district superintendents, 
school principals, external TA providers who work along side SEDL intensive site staff, 
and a sample of teachers, board members, parents, students, and community members. 
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The interviews will be conducted twice each year by the SEDL research team, except 
during the first year, when three rounds of interviews will be conducted. 

Questions will be asked to gather information to learn more about district and school 
personnel's understandings about how to work systemically, the successes they have 
seen to date, where problems have occurred, concerns they have, and other issues that 
address Research Question 2. (A specific interview protocol will be developed in the 
second quarter of FY 01.) Selected software (under exploration at this time) will be used 
to permit entry of interview data into the computer database. 

Data Analysis 

Bimonthly. Every two months, all of the SEDL staff (both intensive site staff and 
research staff) involved in each site will review the site coordinator log. Site Contact 
Records, and available ICM or interview data. With guidance from the SEDL research 
team, the intensive site staff will complete a Site Analysis Form (one form per site). This 
form will synthesize all of the site information into one document. The Site Analysis 
Form will represent the richness of the work at the site and will provide the foundation 
for the construction of procedural knowledge. The following items will focus this set of 
analyses: 

From Site Contact Record, Section A 

• the framework levels, components, and competencies worked on by the site and 

• the strategies implemented by the site, sorted by intent or purpose, and other 
factors. 

From Site Contact Record, Section B 

• indicators that the site is able to work systemically; 

• indicators that the site is becoming a high-performing learning community; 

• the progress or setbacks that the site experienced during the past two-month 
period; 

• supportive conditions that have contributed to the site's progress; 

• barriers that are interfering with the site's progress; and 

• other resources that are used to support the improvement work. 



0 
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In addition, other information related to the context of the site (collected as part 
of recruitment, site entry, data scan, and site exploration) will be considered. A 
description of these is included in Appendix A. 

Semi-Annually. The Site Analysis Forms (backed up by the site coordinator 
logs, Site Contact Records, interview data, available student achievement data, and 
available ICM data) will be synthesized by the intensive site and research staff to 
understand how each site is learning to work systemically to become a high- 
performing learning community, and the contributions and effectiveness of different 
strategies to facilitate such transformation work. These forms will be analyzed to build 
and/or add to displays to organize the data at each site (Miles & Huberman, 1994; 
Patton, 1990; Wolcott, 1994). Multiple displays will be developed to organize and 
format qualitative data (e. g., by event, time, and site dynamics). Miles and Huberman 
(1994) advise that different formats should be tried to answer specific research 
questions. Particular attention will be given in transferring the data to displays to 
identify: 

• the mix of framework levels, components, and competencies on which the site is 
working, and their relationship to the underlying high-performing learning 
community principles; 

• the scope, intensity, and success of the site's systemic work, that is, the site's 
capacity to work systemically; 

• the strategies implemented at the site, sorted by intent and other factors; 

• the movement of the low-performing site to a high-performing learning 
community; and 

• the outcomes of the site's work in terms of increases in reading or mathematics 
achievement by students. 

These data displays will be completed first for individual sites and then across 
district or school sites to help identify patterns of strategies (Miles & Huberman, 1994) 
that influence how districts and schools transform themselves. The data displays will be 
reviewed and discussed by all of the SEDL intensive site and research staff and TA 
providers throughout the year. These discussions will help to verify and triangulate the 
findings. 
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Year 1 Timeline 



August 2001 - 
July 2002 



Logs and Site Contact Records completed after each contact 



Early Fall 
October 2001 



Conduct first round of interviews 
Review logs and Site Contact Records 
Complete Site Analysis Form 
Review logs and Site Contact Records 
Complete Site Analysis Form 
Conduct second round of interviews 
Review logs and Site Contact Records 
Complete Site Analysis Form 
Synthesize and analyze Site Analysis Forms 
Review logs and Site Contact Records 
Complete Site Analysis Form 
Conduct third round of interviews 
Review logs and Site Contact Records 
Complete Site Analysis Form 
Synthesize and analyze Site Analysis Forms 
This schedule will be repeated in subsequent years of the contract, except that only the 
winter and spring rounds of interviews will be conducted in the second through fifth 
years of the contract. 



December 2001 

Early Winter 
February 2002 

April 2002 

Late Spring 
June 2002 

July 2002 



Research Question 3: What knowledge, skills, and tools are needed by 
technical assistance providers to transform low-performing districts and 
schools, and what training is needed to equip them to do this work? 

Rationale 

The technical assistance (TA) infrastructure in each of the five states in SEDL's 
region currently lacks the capacity and/or resources to respond to the numbers of low- 
performing districts and schools requiring assistance. The SEAs in all five states 
acknowledge that there are not enough TA providers available to provide the 
necessary high-quality assistance to low-performing districts and schools. As a result, 
there is a clear need to learn how to equip TA providers to increase the capacity of 
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districts and schools to work systemically and become high-performing learning 
communities. 

Data Collection 

Two primary data collection strategies will be used to answer Research Question 
3. They include an expanded Site Contact Record and interviews. 

Expanded Site Contact Record. An expanded Site Contact Record (described 
initially under Research Question 2) will be the primary tool for data collection for this 
research question, and will be a key method for identifying the knowledge, skills, and 
tools needed by the TA providers and the training needed to equip them to do this 
work. Section A will be expanded to record information not only about any actions(s) 
with the site, but also parallel information about actions(s) with the TA provider at the 
site. 

It is almost certain that a wide variety of knowledge, skills, and tools will be 
required by TA providers, making it difficult to establish in advance a means for 
categorizing or organizing this information. During the first year, SEDL will develop a 
coding system to categorize these to facilitate data collection in future years. A third 
section, Section C, will be added to record information pertaining to Research Question 
3. This section will parallel Section B, as described above in reference to Research 
Question 2. At a minimum, this section will include the knowledge, skills, and tools that 
were employed in this action, why these were used, how effective they seemed to be, 
what else might have been done (or not done), what was the response of the TA 
provider to the action, and what new insights into the training of TA providers were 
gained. 

When the action to be recorded on the Site Contact Record involves SEDL work 
with both a site and an external TA provider, all three sections will be completed. When 
the action to be recorded includes only SEDL work with the external TA provider, then 
only Sections A and C will be completed. For example, a telephone conversation 
regarding work at a study school between a site coordinator and TA provider would be 
recorded in Sections A and C of this form. Section B would be left blank since there are 
no reflections to be recorded about work at the study school for this particular entry. 
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Interviews. During the research team's interviews with teachers, administrators, 
and other individuals (described above related to Research Questions 1.2 and 2), 
questions will be asked about their perspectives on the work of the TA provider. One 
purpose of these questions is to investigate the kinds of knowledge, skills, and tools 
that TA providers used in working with district and school personnel to promote the 
goals of this study. Another purpose is to assess the effectiveness of the efforts of the 
TA providers as perceived by the recipients of those efforts. In doing this, the intent is 
not only to identify the knowledge, skills, and tools that were employed, but how 
useful they were, and what additional knowledge, skills, or tools might have been used 
to make the work of the TA provider even more effective. 

Data Analysis 

Every three months, the site coordinator and the TA provider will engage in an 
in-depth debriefing session to explore and capture in as much detail as possible the 
knowledge, skills, and tools that the TA provider has needed and used, as well as the 
impact they have had. Their discussion will begin with the information from Sections A 
and C of the Site Contact Record. This will permit an analysis of the relationships 
between the framework levels, components, and competencies; the underlying high- 
performing learning community principles; the knowledge, skills, and tools needed by 
the TA provider; and the training or other support offered to the TA providers. In 
addition, the data available from the student achievement analyses, interviews, and 
ICMs will provide rich information about the progress of the sites. The discussion will 
be recorded in a TA Analysis Form, similar to the Site Analysis Form described above; it 
will provide the foundation for the construction of procedural knowledge related to the 
work of TA providers. The following items will focus this set of analyses: 

From Site Contact Record, Section A 

• the framework levels, components, and competencies worked on by the site 
with the TA provider. 

From Site Contact Record, Section B (as relevant) 

• indicators that the site is able to work systemically; 

• indicators that the site is becoming a high-performing learning community; 

• the progress or setbacks that the site experienced recently; 
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• supportive conditions that have contributed to the site's progress; and 

• barriers that are interfering with the site's progress. 

From Site Contact Record, Section C 

• the knowledge, skills, and tools used by the TA provider to work with the site on 
the above; 

• the training strategies that were used to support the development of the TA 
provider; 

• indicators that the strategy or action performed by the TA provider was 
effective; 

• indicators that the training or support to the TA provider was effective; 

• other strategies or actions that might have been performed by the TA provider; 

• response of the site to the TA provider; and 

• response of the TA provider to the training or support provided. 

Semi-Annually. The TA Analysis Forms (backed up by site coordinator logs, 

Site Contact Records, interview data, available student achievement data, and available 
ICM data) will be synthesized by the intensive site and research staff (and TA providers 
whenever feasible) to understand how TA providers are developing the knowledge, 
skills, and tools to support the transformation of low-performing districts and schools. 
Similar to the analyses conducted for Research Question 2, multiple displays will be 
developed to organize and format the qualitative data. Cross-site analyses will be 
conducted to determine if any patterns emerge that relate to the training and work of 
TA providers in particular sites, including: 

• the mix of framework levels, components, and competencies on which the TA 
providers are working with the site, and their relationship to the underlying 
high-performing learning community principles; 

• the strategies implemented by the TA providers at the site, sorted by intent and 
other factors; 

• the knowledge, skills, and tools that TA providers needed to assist the site on this 
work; 

• the strategies used to develop these capacities in TA providers; 

• the scope, intensity, and success of the TA providers with the site; 
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• the TA providers' capacity to help the site; 

• the TA providers' efforts to help transform the low-performing site to a high- 
performing learning community; and 

• the outcome of TA providers' work in the site to increase reading or 
mathematics achievement by students. 

Analyses also will be conducted to determine developmental changes in the TA 
providers' capacity. For example, in the initial phases of the project, are certain 
knowledge, skills, and tools employed more than others in the training and work of TA 
providers and how does this emphasis change over time? Knowing this can be very 
important to the development of a training procedure in this study and in projects 
elsewhere. 

Year 1 Timeline 

August 2001 - 
July 2002 

Three-month 
intervals 

Semi-Annually 

Annual Report 

At the end of each school year, all of the accumulated data for each site will be 
reviewed by the SEDL research team, the site coordinators, and other field staff. This 
review will allow SEDL to understand how districts and schools are transforming across 
the sample of sites and the combinations of specific strategies that contribute to the 
transformation process. The results of this annual review will be summarized in an 
annual report. This report will be submitted to OERI in the first quarter of the 
following year, except for the last year when it will be submitted in the last quarter of 
the contract. 



Record work with TA providers, and training and support 
delivered to TA providers 

Conduct in-depth meetings between the site coordinator and 
the TA provider, approximately three months after the beginning 
of the work at the site and every three months thereafter 

Synthesize and analyze TA Analysis Forms 
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Limitations of the Study 

As with all research, there are particular limitations that should be given 
attention when considering the findings from this study. An obvious first issue is that 
of self-report. The persons who will serve as primary collectors of data will also be 
designing the strategies and actions to support the transformation of sites. These 
individuals will document their own actions as well as those of others at the sites, 
providing the opportunity for perceptions that may be biased or skewed. To mitigate 
against this, SEDL researchers will collect a broad array of data from others at the 
district (e. g., superintendent, district office personnel, community leaders) and school(s) 
(e. g., principal, teachers, parents) whose input can contribute to triangulation, "seeing 
or hearing multiple instances from different sources" (Miles & Huberman, 1994, p. 263). 
This procedure prevents the analysis and acceptance of unverified "initial impressions" 
(Goetz & LeCompte, 1984, p. 11) and enhances the trustworthiness of the data. 
Syntheses of the data every six months also will be submitted to peer reviewers who 
are not a part of the site data collection, but who are knowledgeable and can attest to 
the reasonableness of the findings given the data and their analyses. 

Structural, organizational, or personnel changes at the district and/ or school 
levels may also impact the conduct of this study over the five-year period. Through 
many years of school improvement work, SEDL's experience has shown that such 
changes regularly occur and often disrupt the improvement effort and its impact. 

When such changes occur, they will, of course, be noted in the data collection 
procedures and reports. 

Changes in the state assessment programs may also complicate the conduct of 
this study. Several of the SEAs in SEDL's region are modifying or contemplating 
changes in their specific achievement tests administered or in their testing procedures. 
These changes will complicate comparisons of student results over time. SEDL will note 
these changes when they occur and seek appropriate steps to minimize their impact in 
this study. 



Participation Incentives 

District and school teams will receive a brief summary of the annual synthesis of 
their site's data. Copies of the summary will be shared and discussed with all staff who 
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are participating in the project. SEDL has found that many sites value the opportunity 
to gain feedback and to understand how "things are going." Other incentives for 
participating in the study may be included at the discretion of SEDL and its partners. 

Confidentiality 

SEDL's research policy includes the provision that schools, districts, and/or states 
will not be identified in any way that they may be recognized except by their own staff. 
All field notes and logs, as well as interview information, will be strictly confidential and 
anonymous. If some purpose should develop for identifying a school, district, state, or 
any of its personnel, that will not be done unless permission is obtained from the 
individual, the school, and/or district authority. 




30 



24 



References 



Brophy, J. (1987, April). Educating teachers about managing classrooms and students. 
Paper presented at the annual meeting of the American Educational Research 
Association, Washington, DC. 

Cauley, K. M. (1986, April). Studying knowledge acquisition: Distinctions among 
procedural, conceptual, and logical knowledge. Paper presented at the annual meeting of the 
American Educational Research Association, San Francisco, CA. 

Covey, Steven R. (1989). The seven habits of highly effective people. New York: 

Simon & Schuster, Inc. 

Dentler, R. A. (1984). Putting knowledge to work: Issues in providing effective 
educational dissemination. Phoenix, AZ: Council for Educational Development and 
Research. 

Fullan, M. G. & Miles, M. B. (1992). Getting reform right: What works and what 
doesn't. Phi Delta Kappan, 73(6), 745-752. 

George, A. A., Hall, G. E. & Uchiyama, K. (2000). Extent of implementation of a 
standards-based approach to teaching mathematics and student outcomes. Journal of 
Classroom Interaction, 35(1), 8-25. 

Glaser, R. (1984). Education and thinking: The role of knowledge. American 
Psychologist, 39(2), 93-104. 

Glaser, R. (1992). Expert knowledge and processes of thinking. In D. F. Halpern 
(Ed.), Enhancing thinking skills in the sciences and mathematics (pp. 63-75). Hillsdale, NJ: 
Erlbaum. 

Goetz, J. P. & Le Compte, M. D. (1984). Ethnography and qualitative design in 
educational research. Orlando, FL: Academic Press. 

Hall, G. E. & Hord, S. M. (2001). Implementing change: Patterns, principles and 
potholes. Boston, MA: Allyn and Bacon. 

Kennedy, M. (1983). Working knowledge. Knowledge: Creation, Diffusion, 
Utilization, 5(2), 193-211. 

Kliebard, H. M. (1993). What is a knowledge base, and who would use it if we 
had one? Review of Educational Research, 63(3), 295-303. 

Louis, K.S., & Dentler, R.A. with Rosenblum, S. (1982). Putting knowledge to work: 
An examination of an approach to improvement in education. Cambridge, MA: Abt 
Associates, Inc. 

McCormick, R. (1997). Conceptual and procedural knowledge. International 
Journal of Technology and Design Education, 7, 141-159. 




31 



25 



Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded 
sourcebook. Thousand Oaks, CA : Sage Publications. 

Patton, M. Q. (1990). Qualitative research and evaluation methods. Newbury Park, 
CA: Sage Publications. 

Ryle, G. (1949). The concept of the mind. Harmondsworth, Middlesex, England: 
Penguin Books. 

Wolcott, H. F. (1994). Transforming qualitative data: Description, analysis, and 
interpretation. Thousand Oaks, CA: Sage Publications. 




32 



26 



APPENDIX A 



Demographic and Contextual Information 

SEDL will collect data on each potential site during the recruitment process and 
early work that may be useful in understanding the progress of study districts and 
schools. These data will help describe and explain the environment in which the district 
and school transformations are taking place. In addition, such demographic and 
contextual information will make it possible to select factors for cross-site analyses 
related to the three research questions. 

Pre-Recruitment Data 

These data will be used initially for the purpose of matching sites with the 
selection criteria. Though sites will be somewhat similar in that they will meet the 
established criteria for selection, individual district and school sites may vary a great 
deal. 

Early Site Work Data 

In addition to data collected before entering the sites (noted above), other data 
will be collected in the early intensive work at the sites. 

Site Entry. During this phase, the site coordinator will help create district and 
school leadership teams or encourage the active involvement of existing teams. As part 
of this work, the site coordinators will have the opportunity to observe many aspects of 
the district and schools' operations and to maintain records of these. 

Data Scan. Likewise, in the second phase of SEDL's work in the sites, a data scan 
and the interpretation of its results will be undertaken. In this case, the site coordinator, 
other intensive site staff, and external TA providers will conduct a quick scan to access 
information about staffing patterns, reading and mathematics instructional programs, 
professional development, management, and budgets. The scan will employ interviews 
with the superintendent and central office staff, the principals of schools both in and out 
of the study sample, randomly selected teachers at each grade level, one or more 
members of the school board, and parents and community members. District and 
school records will be examined to reveal the status of the district and schools in terms 
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of their reading and mathematics performance, their readiness to operate as a learning 
community, and an entry point for discussions at the district- and school-levels. These 
data will provide information against which future change and improvement can be 
measured. 

System Exploration. In the third phase of the early intensive site work, system 
exploration, there is another opportunity for data collection. In this set of activities, 
SEDL staff and district and school staff will interact to identify and reflect on the 
"boosters and barriers" that contribute to or detract from student reading and 
mathematics performance. In addition, SEDL expects to stimulate and lead discussions 
about system factors that are available to support the school improvement process, and 
conversely, to note the absence of those factors in the system. 

State Context Data 

SEDL has collected and organized extensive information about each state, 
including standards and state assessments, state accountability systems, and current and 
planned reform issues. This information will be added to the data sets for inclusion in 
the analysis and synthesis of each site's work, and in cross-site analyses. 
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